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ABSTRACT 

A study is proposed that seeks to use the normally 
problematic factor of non-response to a survey in a positive way in 
order estimate certain characteristics of a population subgroup. A 
longitudinal database, the Scientific and Technical Personnel Data 
System (STPDS) measures the educational, demographic, and employment 
characteristics of the nation's scientists and engineers. An 
examination of response trends for STPDS surveys reveals that 
response rates for foreign-born scientists and engineers deteriorate 
at a faster rate over time than do the response rates of those born 
in the United States. It is hypothesised that emigration accounts for 
these differences, and that the difference in the rates of decline of 
response rates can be used as a proxy for emigration of scientists. 
Two very different approaches have been developed and will be used to 
test the hypothesis and to validate the proposed technique. Although 
this methodology is not generalizable to all surveys, it does 
illustrate the importance of a researcher's being thoroughly familiar 
with the survey population and the importance of being open to the 
possibility that challenging and testing the precepts of the practice 
of survey research can result in improvements in methodology. Three 
bar graphs give information about the proposed study. (SLD) 
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Interpreting Nonresoonse in Survey Research: 
Methodological Heresy? 



Introduction 

Of all the potential sources of bias associated with survey 
research, one of the most troubling is nonresponse. Ont indication 
of the degree of seriousness of this problem is the amoun: of 
textbook and journal space devoted to the presentation and testing 
of various survey techniques that can oe used to stimulate response. 
Another indication of the significance of the problem is the 
attention given to measuring the bias associated with nonresponse. 
Although the true effect of actual nonresponse cannot be known, the 
literature is replete with alternative ways of evaluating its 
probable effect on parameter estimates. 

It is noi the purpose of the present paper to review the various 
approaches that have be§n proposed to decrease nonresponse or to 
evaluate or estimate the' direction and/or magnitude of the biasei 
associated with nonresponse. The reader who is interested in 
exploring alternative techniques for accomplishing such objectives 
is referred elsewhere--for example, to an excellent three-volume 
report issued by the Panel on Incomplete Data, Committee on 
National Statistics, National Research Council (Madow et al., 1983), 
which includes a rather comprehensive bibliography covering many 
nonresponse issues. Additional general coverage can be found in a 
well-organized but somewhat outdated publication by the Census 
Bureau (U.S. Bureau of the Census, 1974). 
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The present paper focuses on a study that seeks to use the 
normally problematic factor of survey nonresponse in a positive 
way--to estimate certain characteristics of a population subgroup. 
The specific context in which the technique is applied-within the 
framework of a longitudinal survey-is one that admittedly may not 
be applicable to many educational researchers. Even those who do 
not use longitudinal techniques may find the present study to be 
instructive, however, because it represents an example of an 
innovative solution to a significant problem, made possible by the 
investigators' familiarity wit'.i their survey population-certainly a 
lesson that can benefit all survey researchers. 

The Nature of the Problem 

Context 

The National Science Foundation Act of 1950, as amended by Title 
42, United States Code, Section 1862, mandates that the National 
Science Foundation (NSF) obtain and provide current information 
about the nation's scientists and engineers. One of the primary 
mechanisms by which NSF fulfills this obligation is by maintaining, 
updating, and improving a longitudinal data base, the Scientific and 
Technical Personnel Data System (STPDS), which is designed to 
measure the educational, demographic, and employment 
characteristics of the nation's scientists and engineers. (Details 
about the three individual surveys which comprise the STPDS and 
how they are integrated to produce national estimates are not 
relevant to the purposes of this paper. The reader who is interested 
in learning more about the technical characteristics of this data 
base is referred to NSF's Guide to Science/Engineering Data .^ 

U.S. reliance on foreign-born scientists and engineers in certain 
disciplines— particularly those individuals holding advanced 
degrees-is well-established (Finn, 1985; Finn, 1987; National 
Research Council, 1988). Monitoring the flows (i.e., immigration and 
emigration) of these professionals, therefore, is important for 
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assessing the size and characteristics of the population of 
scientists and engineers in the United States. Unfortunately, during 
the past 10 or 15 years, the resources devoted to the careful 
tracking of immigration and emigration of aii kinds have declined 
significantly in this country (Levine, Hill, & Warren, 19S5). As a 
result, innovative, low-cost techniques for estimating the number 
and movement of foreign-born scientists and engineers into and out 
of the United Spates are needed. Although both sides of the science 
and engineering personnel flow "equation" (in-flows as immigrants 
and out-flows as emigrants) are important, it is the emigration 
component that is *he focus of the present paper. 

Emigration data are not routinely collect by the U.S. 
government. If such data were collected tnty would probably be of 
limited value, because persons leaving the United States may not 
give accurate answers to questions about their intentions to return. 
Thus, it is very difficult to distinguish the emigrant from the person 
who is leaving only temporarily for vacation or work. Given the 
extreme difficulty of measuring emigration in general, measuring 
that of a subgroup of the population-scientists and engineers in 
this case-becomes even more Difficult. 

Problems Associated With Survey Nonresponse in the Target 
Population 

Emigration is not taken into account . It is thought that many 
immigrant scientists and engineers do not stay in the United States 
until they retire (Finn, 1985). To the extent that they do in fact 
emigrate after a short stay in this country, the STPDS is designed in 
such a way that an overestimate of the number of foreign-born 
scientists and engineers in the United States is almost certain to 
result. This overestimate arises from the way survey 
nonrespondents are treated in obtaining population estimates: the 
weights of the respondents are increased to compensate for the 
nonrespondents in a given stratum, so that the number of weighted 
observations in a stratum cannot be affected by nonresponse alone. 



The estimation procedure is a "closed system" in this regard; in 
effect, it is not possible for a nonrespondent to leave the target 
population. 

Because the STHDS is a longitudinal data base and because the 
sampling and stratification involved in the constituent surveys are 
very complicated, the nature and magnitude of biases introduced 
through such estimation procedures are impossible to predict for 
particular characteristics of groups meeting specified demographic 
criteria. If foreign-born immigrants tend to emigrate during the 
period of the survey, the "closed" nature of the system will lead to 
observations with higher and higher weights for those who si£ 
respond. The use of these large-weight observations in analyses 
involving relatively few raw observations consequently become less 
and less reliable. 

Erroneous conclusions mav be drawn . The effect of sample 
weights for respondents increasing over time can be illustrated in 
some data (based on a weighted analysis of one nf the surveys which 
make up the STPDS) that were presented to support the contention 
that engineering schools i r U.S. universities are increasingly relying 
on foreign-born engine' j to fill new faculty vacancies (See Figure 
1). Though it is ce .mly possible that the trend in Figure 1 
represents the true otate of affairs, it is far more likely that the 
magnitude of the change depicted is exaggerated by the treatment of 
nonresponding immigrants described above, since there was a large 
rise in the non response of- foreign-born members of the survey 
population during the years considered in the analysis. 

Hypothesis 

An examination of response trends for the relevant STPDS 
surveys reveals that the response rates o: foreign-born scientists 
and engineers deteriorate at a faster rate over time than do the 
response rates of U.S.-born scientists and engineers. For example, 
Figure 2 shows tho overall fall-off in response rates for the Survey 



of Doctorate Recipients (one of the STPDS surveys) between 1981 
and 1983 and between 1981 and 1985 for various citizenship 
classifications. It is important to note that all of the individuals 
represented in this figure responded to the 1981 survey. 

Figure 3 depicts an even more dramatic difference among 
citizenship categories and is perhaps even more relevant to the 
issue at hand. Whereas Figure 2 includes responses that were 
received from foreign as well as U.S. addresses, Figure 3 includes 
only those responses received from U.S. addresses. It is plausible 
that the responses from foreign addresses are much more likely to 
include individuals who will not return to the United States and are, 
therefore, not appropriately considered to be part of the stock of 
U.S. scientists and engineers. 

We pre nose that the differential decline in response 
rates bet\ r een U.S.-born and foreign-born scientists and 
engineers be attributed to emigration and that the 
difference in the rate of decline be used as a proxy for 
emigration. Let us first examine the ration^ !e for this 
interpretation and then suggest procedures for testing the 
hypothesis. 

Rationale for Nonresponse Interpretation 

Control for initial diffe rences in attitu d es towards surveys . The 
most convincing explanations one hears when considering the 
response rate differences that commonly occur between U.S.-born 
and foreign-born survey recipients seem to focus on such ingrained 
personal characteristics as cultural differences and different 
senses of obligation (or suspicion) associated with government 
surveys. In the present context these differences are in some sense 
"controlled for" by starting with a group of individuals all of whom 
responded to the survey in the base year (e.g., 1981 for the Survey of 
Doctorate Recipients). Though there are many reasons that could 
account for the observed differences in response rate deterioration 



between U.S.-born and foreign-born scientists and engineers, the 
authors contend that the most important of these other causes can 
account for only a trivial difference between the two groups, 
because the willingness-to-respond "control" lends more credence to 
the emigration interpretation. 

Relative propensity to emigrate . Finn (1985) found that fewer 
than 20 percent of foreign-born scientists and engineers came to the 
United States at an early enough age to attend secondary school here. 
Most of the foreign-born scientists and engineers were educated and 
socialized in foreign countries, and often they received 
undergraduate college degrees from foreign universities. They are 
fluent in foreign languages and are able to function as natives in 
other cultures. As such, they are much more employable in foreign 
countries than is the average U.S.-born scientist or engineer. The 
foreign-born are also more likely to have relatives abroad. Some 
foreign-born scientists and engineers are obligated to return home 
and/or have difficulty staying in the United States. For these and 
for other reasons, the foreign-born scientists and engineers in the 
United States are likely to have a positive emigration rate. 

The U.S.-born leave the United States too, but mostly for 
temporary assignments in foreign countries. Available evidence 
does not support the view that there is substantial net emigration of 
U.S.-born scientists and engineers. Consider the case of the United 
Kingdom, which, because of language and the prestige of its 
scientific institutions, may be the most popular destination for U.S. 
scientists and engineers leaving the United States. A recent study 
by the Royal Society (1987) identified by name 685 science or 
engineering immigrants to the United Kingdom over the past ten 
years. Of these, only 76 (11 percent) were from North America, 
some of whom we may assume were from Canada. Further, the Royal 
Society reports that most of those counted as arriving during the 
preceding ten years were no longer in the United Kingdom at the time 
of the survey. Ignoring the fact that some must be from Canada and 
that some may not have doctorates, the immigration recorded by the 
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Royal Society could have been produced by a ten-year emigration 
rate of 0.01 percent of the U.S. population of doctoral scientists and 
engineers. 

Of course, emigration from the United States to the United 
Kingdom is only part of the total U.S. emigration of scientists and 
engineers. We Know, however, that the United Kingdom is one of the 
most popular sites for Americans who go abroad for scientific visits 
(Finn & Sigmon, 1988). It seems likely, therefore, that any 
permanent emigration of U,S.-born scientists and engineers would 
involve the United Kingdom at least as frequently. If this 
assumption is accepted, we could extrapolate from the Royal Society 
study to conclude that emigration of U.S.-born scientists and 
engineers is probably less than ten times the emigration to the 
United Kingdom, and thus that the total emigration rate tor this 
group is probably no more than 0.1 percent. 

Becoming acculturated . In general terms, one would expect that 
the longer a foreign-born person stays in the United States, the more 
he/she would tend to adopt the attitudes and behaviors of U.S. 
natives. That being the case, one would expect naturalized U.S. 
citizens to act more like U.S. natives than non-U.S. citizen 
D°rmanent residents. Likewise one would expect non-U.S. citizen 
permanent residents to behave more like U.S. natives than non-U.S. 
citizen temporary residents. In terms of response to the 1981 
Survey of Doctorate Recipients, this is generally the case. Making no 
assumptions about the relative likelihood of actually being 
contacted; the 1981 response rates for U.S. natives, naturalized U.S. 
citizens, non-U.S. citizen permanent residents, and non-U.S. citizen 
temporary residents were 69%, 7b%, 36%, and 8% respectively. 
(Bear in mind, however, that this has no relationship to Figures 2 
and 3.) 

As far as the likelihood of an individual's leaving this country is 
concerned, we would expect the same relationship to hold: we would 
expect the groups that have more ties to the United States (e.g., 



career, financial investments, friends, family, interests) to exhibit 
a lower propensity to emigrate than those who have fewer of those 
ties. As we have seen above, very few U.S. natives emigrate. 
Figures 2 and 3 reflect the patterns that one would expect with 
regard to the other citizenship classifications. 

Rival hypotheses . Why might a person respond to the survey in 
1981 but not in 1985? Many possibilities exist, but several major 
reasons merit consideration: he/she (1) died; (2) retired; (3) moved 
within the United States and the questionnaire was not forwarded; 
(4) simply refused to cooperate; (5) left the United States and the 
questionnaire was not forwarded; (6) received the questionnaire in a 
foreign land but thought a response might not be needed from a 
person now outside the United States; or (7) received the 
questionnaire in a foreign land, could not use the business reply 
envelope, and did not want to use personal funds to pay the return 
postage. 

!t would be difficult to argue that the foreign-born scientist or 
engineer who had responded in 1981 would be any less likely to 
respond than the U.S. natives on any one of the first four counts 
listed. In fact, with regard to death and retirement, just the 
opposite would more likely be the case. It has been shown that 
foreign-born scientists and engineers tend to be younger than their 
U.S.-born counterparts (Finn, 1985). Migration within the United 
States may differ slightly between the two groups, but no data are 
available to support this. Most importantly, the authors would argue 
that there is no reason to expert a higher or lower refusal rate for 
the foreign-born. As previously stated, there may be a reason for 
the foreign-born to be more or less likely to respond to surveys like 
this in the first place, but the sample on which Figures 2 and 3 are 
based all responded in 1981. This leaves reasons 5, 6, and 7 listed 
above. Any person who emigrates is less likely to have his/her mail 
forwarded and is less likely to respond if the questionnaire should 
reach him/her. 



Consistency of observed patterns with expectations . The 
patterns of response fall-off that were observed are consistent 
with the above rationale. There is a greater fall-off for non-U.S. 
natives than for U.S. natives; the fall-off for temporary non-citizens 
is greater than that for permanent non-citizens; and the fall-off for 
non-native citizens is groater than that for native citizens. 

Validating the Hypothesis 

Because the observed patterns of response deterioration could be 
explained logically for the Survey of Doctorate Recipients, because 
these patterns held up across the other surveys comprising the 
STPDS, because there was convincing evidence that the current 
estimation procedures were not adequate, and because no other 
acceptable alternative had been developed, the argument outlined 
above was used to develop preliminary estimates of the emigration 
of foreign-born scientists and engineers (Finn & Clark, 1988). 
Inasmuch as the process is more germane to the present purpose 
than are the outcomes, no space will be devoted to these estimates 
in this paper. It should be noted, however, that one of the ancillary 
benefits of this estimation procedure was to decrease the effective 
nonresponse rate for the foreign-born (since those nonrespondents 
who were classified as emigrants in the algorithm were no longer 
legitimately considered a member of the target population). 

Inasmuch as the issues raised in the course of the present study 
have significant implications for the integrity of NSPs Scientific 
and Technical Personnel Data System and the reliability of the 
estimates of the stock of U.S. scientists and engineers that are 
produced from it, NSF has been supportive of the authors* efforts to 
develop empirical' validation of the proposed estimation procedure. 
Two very different approaches have been developed to test the 
hypothesis, one of which can be used on one of the STPDS surveys 
and one of which can be used on two of the surveys. Once all three 
studies have been carried out, the validity of the proposed technique 
can be assessed. 



Importance of the Study for Survey Researchers 

The authors recognize that the methodology presented in this 
paper is not generalizable (at least in a "wholesale- manner) to 
surveys in general, or even to longitudinal surveys. They do believe, 
however, that this study holds several lessons for ail survey 
researchers. 

First, it underscores the advantages of a researcher's being 
thoroughly familiar with his/her survey population. Also, it 
suggests the value of keeping track of the general characteristics of 
those who respond to a survey. It is probably a very rare case when 
an investigator knows nothing about the individuals in his/her 
survey population and when it is not advisable or not possible to 
include an identification code so that incoming responses can be 
tracked. The information that he/she has on the nonrespondents may 
be useful in gaining some insight into how the respondents compare 
to the nonrespondents on a few gross demographic variables perhaps. 
Such an analysis may or may not offer any clues as to what sorts of 
nonsampling bias may exist, but they might suggest alternative 
contact or follow-up procedures for present or future use. 

More importantly, it is hoped that this paper will encourage 
survey researchers to accept the "tenets- regarding the practice of 
survey research with at least a modicum of skepticism, because 
challenging and testing these precepts can. lead to improvements in 
methodology. The researcher who is haunted by the feeling that 
his/her survey application is somehow unique and that mavbe- iust 
mavbe- -all of the "golden rules" do not apply, may not be guilty of 
heresy but only of critical, creative thinking that may eventually 
lead to the development of solutions to "unsolvable" problems. 
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FIGURE 1. FOREIGN-BORN AS A PROPORTION C- ALL ENGINEERING FACULTY, 
AGED 35 OR LESS 
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SOURCE: Based on special tabulations Srom the National Research Council. 

FIGURE 2. CHANGES IN TOTAL RESPONSE RATES BY CITIZENSHIP STATUS 
FOR THOSE INDIVIDUALS WHO RESPONDED TO THE 1981 SURVEY 
OF DOCTORATE RECIPIENTS. 
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FIGURE 3. CHANGES IN U.S. RESPONSE RATES BY CITIZENSHIP STATUS 

FOR THOSE INDIVIDUALS WHO RESPONDED TO THE 1981 SURVEY 
OF DOCTORATE RECIPIENTS. 



